Sample-efficient multi-agent reinforcement learning with masked reconstruction.

Deep reinforcement learning (DRL) is a powerful approach that combines reinforcement learning (RL) and deep learning to address complex decision-making problems in high-dimensional environments. Although DRL has been remarkably successful, its low sample efficiency necessitates extensive training times and large amounts of data to learn optimal policies. These limitations are more pronounced in the context of multi-agent reinforcement learning (MARL). To address these limitations, various studies have been conducted to improve DRL. In this study, we propose an approach that combines a masked reconstruction task with QMIX (M-QMIX). By introducing a masked reconstruction task as an auxiliary task, we aim to achieve enhanced sample efficiency-a fundamental limitation of RL in multi-agent systems. Experiments were conducted using the StarCraft II micromanagement benchmark to validate the effectiveness of the proposed method. We used 11 scenarios comprising five easy, three hard, and three very hard scenarios. We particularly focused on using a limited number of time steps for each scenario to demonstrate the improved sample efficiency. Compared to QMIX, the proposed method is superior in eight of the 11 scenarios. These results provide strong evidence that the proposed method is more sample-efficient than QMIX, demonstrating that it effectively addresses the limitations of DRL in multi-agent systems.

STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari.

With the development of deep learning technology, deep reinforcement learning (DRL) has successfully built intelligent agents in sequential decision-making problems through interaction with image-based environments. However, learning from unlimited interaction is impractical and sample inefficient because training an agent requires many trial and error and numerous samples. One response to this problem is sample-efficient DRL, a research area that encourages learning effective state representations in limited interactions with image-based environments. Previous methods could effectively surpass human performance by training an RL agent using self-supervised learning and data augmentation to learn good state representations from a given interaction. However, most of the existing methods only consider similarity of image observations so that they are hard to capture semantic representations. To address these challenges, we propose spatio-temporal and action-based contrastive representation (STACoRe) learning for sample-efficient DRL. STACoRe performs two contrastive learning to learn proper state representations. One uses the agent's actions as pseudo labels, and the other uses spatio-temporal information. In particular, when performing the action-based contrastive learning, we propose a method that automatically selects data augmentation techniques suitable for each environment for stable model training. We train the model by simultaneously optimizing an action-based contrastive loss function and spatio-temporal contrastive loss functions in an end-to-end manner. This leads to improving sample efficiency for DRL. We use 26 benchmark games in Atari 2600 whose environment interaction is limited to only 100k steps. The experimental results confirm that our method is more sample efficient than existing methods. The code is available at https://github.com/dudwojae/STACoRe.

Uncertainty-aware hierarchical segment-channel attention mechanism for reliable and interpretable multichannel signal classification.

Multichannel signal data analysis has been crucial in various industrial applications, such as human activity recognition, vehicle failure predictions, and manufacturing equipment monitoring. Recently, deep neural networks have come into use for multichannel signal data because of their ability to automatically extract useful features from complex multichannel signals. However, deep neural networks are black-box models whose internal working mechanisms cannot be put in a form readily understood by humans. To address this issue, we have proposed an uncertainty-aware hierarchical segment-channel attention model that consists of a time segment and channel level attentions. The hierarchical attention mechanism enables a neural network to identify important time segments and channels critical for prediction, making the model explainable. In addition, the model uses variational inferences to provide uncertainty information that yields a confidence interval that can be easily explained. We conducted experiments on simulated and real-world datasets to demonstrate the usefulness and applicability of our method. The results confirm that our method can attend to important time segments and sensors while achieving better classification performance.

3-Dimensional convolutional neural networks for predicting StarCraft â ¡ results and extracting key game situations.

In real-time strategy games, players collect resources, control various units, and create strategies to win. The creation of winning strategies requires accurately analyzing previous games; therefore, it is important to be able to identify the key situations that determined the outcomes of those games. However, previous studies have mainly focused on predicting game results. In this study, we propose a methodology to predict outcomes and to identify information about the turning points that determine outcomes in StarCraft Ⅱ, one of the most popular real-time strategy games. We used replay data from StarCraft Ⅱ that is similar to video data providing continuous multiple images. First, we trained a result prediction model using 3D-residual networks (3D-ResNet) and replay data to improve prediction performance by utilizing in-game spatiotemporal information. Second, we used gradient-weighted class activation mapping to extract information defining the key situations that significantly influenced the outcomes of the game. We then proved that the proposed method outperforms by comparing 2D-residual networks (2D-ResNet) using only one time-point information and 3D-ResNet with multiple time-point information. We verified the usefulness of our methodology on a 3D-ResNet with a gradient class activation map linked to a StarCraft Ⅱ replay dataset.

A Machine Learning-Based Study of the Effects of Air Pollution and Weather in Respiratory Disease Patients Visiting Emergency Departments.

BACKGROUND: To date, investigating respiratory disease patients visiting the emergency departments related with fined dust is limited. This study aimed to analyze the effects of two variable-weather and air pollution on respiratory disease patients who visited emergency departments. METHODS: This study utilized the National Emergency Department Information System (NEDIS) database. The meteorological data were obtained from the National Climate Data Service. Each weather factor reflected the accumulated data of 4 days: a patient's visit day and 3 days before the visit day. We utilized the RandomForestRegressor of scikit-learn for data analysis. RESULT: The study included 525,579 participants. This study found that multiple variables of weather and air pollution influenced the respiratory diseases of patients who visited emergency departments. Most of the respiratory disease patients had acute upper respiratory infections [J00-J06], influenza [J09-J11], and pneumonia [J12-J18], on which PM10 following temperature and steam pressure was the most influential. As the top three leading causes of admission to the emergency department, pneumonia [J12-J18], acute upper respiratory infections [J00-J06], and chronic lower respiratory diseases [J40-J47] were highly influenced by PM10. CONCLUSION: Most of the respiratory patients visiting EDs were diagnosed with acute upper respiratory infections, influenza, and pneumonia. Following temperature, steam pressure and PM10 had influential relations with these diseases. It is expected that the number of respiratory disease patients visiting the emergency departments will increase by day 3 when the steam pressure and temperature values are low, and the variables of air pollution are high. The number of respiratory disease patients visiting the emergency departments will increase by day 3 when the steam pressure and temperature values are low, and the variables of air pollution are high.

Generative Adversarial Networks for De Novo Molecular Design.

In the chemical industry, the generation of novel molecular structures with beneficial pharmacological and physicochemical properties in de novo molecular design is a critical problem. The advent of deep learning and neural generative models has recently enabled significant achievements in constructing molecular design models in de novo design. Consequently, studies on new generative models continue to generate molecules that exhibit more useful chemical properties. In this study, we propose a method for de novo design that utilizes generative adversarial networks based on reinforcement learning for realistic molecule generation. This method learns to reproduce the training data distribution of simplified molecular-input line-entry system strings. The proposed method is demonstrated to effectively generate novel molecular structures from five benchmark results using a real-world public dataset, ChEMBL. The code is available at https://github.com/dudwojae/SMILES-MaskGAN.

Analysis of the Adequacy of Prehospital Emergency Medical Services Use of Patients Who Visited Emergency Departments in Korea from 2016 to 2018: Data from the National Emergency Department Information System.

INTRODUCTION: Proper ambulance use is important not only due to the patient's transport quality but also because of the need for efficient use of limited resources allotted by the system. Therefore, this study was conducted to check for overuse or underuse of the ambulance system by patients who visited the emergency department (ED). METHODS: In this study, a secondary data analysis was conducted using the existing database of the National Emergency Department Information System with all patients who visited EDs over the three-year study period from 2016 to 2018. The study subjects were classified into the following groups: (1) appropriate Emergency Medical Services (EMS) usage; (2) appropriate no EMS usage; (3) underuse; and (4) overuse groups. RESULTS: Of 18,298,535 patients, 11,668,581 (63.77%) were classified under the appropriate usage group, while 6,629,954 (36.23%) were classified under the inappropriate usage group. In the appropriate EMS usage group, there were 2,408,845 (13.16%) patients. In the appropriate no EMS usage group, there were 9,259,706 (50.60%) patients. As for the inappropriate usage group, there were 5,147,352 (28.13%) patients categorized under the underuse group. On the other hand, there were 1,482,602 (8.10%) patients under the overuse group. CONCLUSION: There are many patients who use ambulances appropriately, but there are still many overuse and underuse. Guidelines on ambulance use are necessary for the efficient use of emergency medical resources and for the safety of patients.

Multi-agent reinforcement learning with approximate model learning for competitive games.

We propose a method for learning multi-agent policies to compete against multiple opponents. The method consists of recurrent neural network-based actor-critic networks and deterministic policy gradients that promote cooperation between agents by communication. The learning process does not require access to opponents' parameters or observations because the agents are trained separately from the opponents. The actor networks enable the agents to communicate using forward and backward paths while the critic network helps to train the actors by delivering them gradient signals based on their contribution to the global reward. Moreover, to address nonstationarity due to the evolving of other agents, we propose approximate model learning using auxiliary prediction networks for modeling the state transitions, reward function, and opponent behavior. In the test phase, we use competitive multi-agent environments to demonstrate by comparison the usefulness and superiority of the proposed method in terms of learning efficiency and goal achievements. The comparison results show that the proposed method outperforms the alternatives.

Analysis of bacterial FAMEs using gas chromatography - vacuum ultraviolet spectroscopy for the identification and discrimination of bacteria.

The identification of microorganisms is very important in different fields and alternative methods are necessary for a rapid and simple identification. The use of fatty acids for bacterial identification is gaining attention as phenotypic characteristics are reflective of the genotype and are more easily analyzed. In this work, gas chromatography-vacuum ultraviolet spectroscopy (GC-VUV) was used to determine bacteria fatty acid methyl esters (FAMEs), to identify and discriminate different environmental bacteria based on their fatty acid profile. Microorganisms were grown in agar and their fatty acids extracted, saponified, and esterified before analysis. Unique FAME profiles were obtained for each microorganism mainly composed of branched, cyclopropane, hydroxy, saturated, and unsaturated fatty acid methyl esters. S. maltophilia showed a higher diversity of fatty acids while Bacillus species showed higher complexity in terms of branched-chain FAMEs, with several iso and anteiso forms. 12 different bacteria genera and 15 species were successfully differentiated based on their fatty acid profiles after performing PCA and cluster analysis. Some difficult to differentiate species, such as Bacillus sp., which are genetically very similar, were differentiated with the developed method.

Dental informatics to characterize patients with dentofacial deformities.

Relevant statistical modeling and analysis of dental data can improve diagnostic and treatment procedures. The purpose of this study is to demonstrate the use of various data mining algorithms to characterize patients with dentofacial deformities. A total of 72 patients with skeletal malocclusions who had completed orthodontic and orthognathic surgical treatments were examined. Each patient was characterized by 22 measurements related to dentofacial deformities. Clustering analysis and visualization grouped the patients into three different patterns of dentofacial deformities. A feature selection approach based on a false discovery rate was used to identify a subset of 22 measurements important in categorizing these three clusters. Finally, classification was performed to evaluate the quality of the measurements selected by the feature selection approach. The results showed that feature selection improved classification accuracy while simultaneously determining which measurements were relevant.

Association rule mining and network analysis in oriental medicine.

Extracting useful and meaningful patterns from large volumes of text data is of growing importance. In the present study we analyze vast amounts of prescription data, generated from the book of oriental medicine to identify the relationships between the symptoms and the associated medicines used to treat these symptoms. The oriental medicine book used in this study (called Bangyakhappyeon) contains a large number of prescriptions to treat about 54 categorized symptoms and lists the corresponding herbal materials. We used an association rule algorithm combined with network analysis and found useful and informative relationships between the symptoms and medicines.

Sulfur amino acid-free diet results in increased glutamate in human midbrain: a pilot magnetic resonance spectroscopic study.

OBJECTIVE: This pilot study was designed to determine if metabolic effects in different brain regions (left and right parietal lobes, midbrain) caused by 3 d of food consumption without methionine or cysteine could be detected by proton magnetic resonance spectroscopy. METHODS: Healthy individuals 18 to 36 y old (n = 8) were studied by magnetic resonance spectroscopy after receiving a diet with adequate sulfur amino acids (SAAs) or with zero SAA for 3 d. Pulse sequences were used to selectively measure glutathione (GSH), and linear combination modeling of spectra was used to measure other high-abundance brain metabolites and expressed relative to creatine (Cr). RESULTS: Although dietary SAAs are required to maintain GSH, the 3-d SAA insufficiency resulted in no significant change in GSH/Cr in the three brain regions. Principal component analysis of 16 metabolites measured by linear combination modeling showed that the metabolic pattern in the midbrain, but not in the parietal lobes, was distinguished according to the dietary SAAs. Multivariate statistical analysis showed that the major discriminating factors were signals of glutamate/Cr, (glutamate + glutamine)/Cr, and myoinositol/Cr. Correlation analyses between midbrain metabolites and GSH-related metabolites in plasma showed that midbrain glutamate/Cr had an inverse correlation with plasma cystine. CONCLUSION: The data show that magnetic resonance spectroscopy is a non-invasive tool suitable for nutritional assessment and suggest that nutritional imbalance caused by 3 d of SAA-free food more selectively affects the midbrain than the parietal lobes.

Matrix-assisted laser desorption/ionization-mass spectrometry of cuticular lipid profiles can differentiate sex, age, and mating status of Anopheles gambiae mosquitoes.

Malaria is a devastating mosquito-borne disease, which affects hundreds of millions of people each year. It is transmitted predominantly by Anopheles gambiae, whose females must be >10 days old to become infective. In this study, cuticular lipids from a laboratory strain of this mosquito species were analyzed using a mass spectrometry method to evaluate their utility for age, sex and mating status differentiation. Matrix-assisted laser desorption/ionization-mass spectrometry (MALDI-MS), in conjunction with an acenaphthene/silver nitrate matrix preparation, was shown to be 100% effective in classifying A. gambiae females into 1, 7-10, and 14 days of age. MALDI-MS analysis, supported by multivariate statistical methods, was also effective in detecting cuticular lipid differences between the sexes and between virgin and mated females. The technique requires further testing, but the obtained results suggest that MALDI-MS cuticular lipid spectra could be used for age grading of A. gambiae females with precision greater than with other available methods.

Matrix-assisted laser desorption/ionization-time of flight-mass spectrometry profiling of trace constituents of condom lubricants in the presence of biological fluids.

The use of condoms in sexual assault cases has become increasingly common due to the heightened awareness of the use of DNA as evidence in criminal investigations. The ability to identify and differentiate the polymers and additives found in lubricant residues can provide investigators leads and insights as to the perpetrator of a sexual assault. Matrix-assisted laser desorption/ionization-time of flight-mass spectrometry (MALDI-TOF-MS) is ideal for detecting condom lubricants and additives; the instrument is capable of surveying analytes across a wide mass range and is a preferred technique for the analysis of polymers. Three MALDI-TOF-MS methods directed toward the detection and differentiation of condom and personal lubricant residues, as well as their mixtures with biological fluids, were developed and compared: (a) a sample premixed with aqueous matrix; (b) a sample premixed with an ionic liquid matrix; and (c) a layering method that incorporates a cationization reagent. Of the three, the layered method that utilized sodium chloride as a cationization reagent showed the best sensitivity and selectivity. This method allowed for the segregation of the various lubricant formulas into a discrete number of groups. Infrared spectroscopy was used to support and clarify the MALDI data. Principal component analysis was used to further demonstrate the ability of this method to segregate various lubricant types into a limited number of classes. Additionally, lubricant residues could be detected in the presence of biological fluids down to a fraction of a percent.

A data-integrated simulation-based optimization for assigning nurses to patient admissions.

The health care system in the United States has a shortage of nurses. A careful planning of nurse resources is needed to ease the health care system from the burden of the nurse shortage and standardize nurse workload. An earlier research study developed a data-integrated simulation to evaluate nurse-patient assignments (SIMNA) at the beginning of a shift based on a real data set provided by a northeast Texas hospital. In this research, with the aid of the same SIMNA model, two policies are developed to make nurse-to-patient assignments when new patients are admitted during a shift. A heuristic (HEU) policy assigns a newly-admitted patient to the nurse who has performed the least assigned direct care among all the nurses. A partially-optimized (OPT) policy seeks to minimize the difference in workload among nurses for the entire shift by estimating the assigned direct care from SIMNA. Results comparing HEU and OPT policies are presented.

A data-integrated simulation model to evaluate nurse-patient assignments.

This research develops a novel data-integrated simulation to evaluate nurse-patient assignments (SIMNA) based on a real data set provided by a northeast Texas hospital. Tree-based models and kernel density estimation (KDE) were utilized to extract important knowledge from the data for the simulation. Classification and Regression Tree models, data mining tools for prediction and classification, were used to develop five tree structures: (a) four classification trees from which transition probabilities for nurse movements are determined, and (b) a regression tree from which the amount of time a nurse spends in a location is predicted based on factors such as the primary diagnosis of a patient and the type of nurse. Kernel density estimation is used to estimate the continuous distribution for the amount of time a nurse spends in a location. Results obtained from SIMNA to evaluate nurse-patient assignments in Medical/Surgical unit I of the northeast Texas hospital are discussed.

Individual variation in macronutrient regulation measured by proton magnetic resonance spectroscopy of human plasma.

Proton nuclear magnetic resonance ((1)H-NMR) spectroscopy of plasma provides a global metabolic profiling method that shows promise for clinical diagnostics. However, cross-sectional studies are complicated by a lack of understanding of intraindividual variation, and this limits experimental design and interpretation of data. The present study determined the diurnal variation detected by (1)H NMR spectroscopy of human plasma. Data reduction methods revealed three time-of-day metabolic patterns, which were associated with morning, afternoon, and night. Major discriminatory regions for these time-of-day patterns included the various kinds of lipid signals (-CH(2)- and -CH(2)OCOR), and the region between 3 and 4 ppm heavily overlapped with amino acids that had alpha-CH and alpha-CH(2). The phasing and duration of time-of-day patterns were variable among individuals, apparently because of individual difference in food processing/digestion and absorption and clearance of macronutrient energy sources (fat, protein, carbohydrate). The times of day that were most consistent among individuals, and therefore most useful for cross-sectional studies, were fasting morning (0830-0930), postprandial afternoon (1430-1630), and nighttime samples (0430-0530). Importantly, the integrated picture of metabolism provided by (1)H-NMR spectroscopy of plasma suggests that this approach is suitable to study complex regulatory processes, including eating patterns/eating disorders, upper gastrointestinal functions (gastric emptying, pancreatic, biliary functions), and absorption/clearance of macronutrients. Hence, (1)H-NMR spectroscopy of plasma could provide a global metabolic tolerance test to assess complex processes involved in disease, including eating disorders and the range of physiological processes causing dysregulation of energy homeostasis.

Discovery of metabolite features for the modelling and analysis of high-resolution NMR spectra.

This study presents three feature selection methods for identifying the metabolite features in nuclear magnetic resonance spectra that contribute to the distinction of samples among varying nutritional conditions. Principal component analysis, Fisher discriminant analysis, and Partial Least Square Discriminant Analysis (PLS-DA) were used to calculate the importance of individual metabolite feature in spectra. Moreover, an Orthogonal Signal Correction (OSC) filter was used to eliminate unnecessary variations in spectra. We evaluated the presented methods by comparing the ability of classification based on the features selected by each method. The result showed that the best classification was achieved from an OSC-PLS-DA model.

Characterization of spatially homogeneous regions based on temporal patterns of fine particulate matter in the continental United States.

Statistical analyses of time-series or spatial data have been widely used to investigate the behavior of ambient air pollutants. Because air pollution data are generally collected in a wide area of interest over a relatively long period, such analyses should take into account both spatial and temporal characteristics. The objective of this study is 2-fold: (1) to identify an efficient way to characterize the spatial variations of fine particulate matter (PM2.5) concentrations based solely upon their temporal patterns, and (2) to analyze the temporal and seasonal patterns of PM2.5 concentrations in spatially homogenous regions. This study used 24-hr average PM2.5 concentrations measured every third day during a period between 2001 and 2005 at 522 monitoring sites in the continental United States. A k-means clustering algorithm using the correlation distance was used to investigate the similarity in patterns between temporal profiles observed at the monitoring sites. A k-means clustering analysis produced six clusters of sites with distinct temporal patterns that were able to identify and characterize spatially homogeneous regions of the United States. The study also presents a rotated principal component analysis (RPCA) that has been used for characterizing spatial patterns of air pollution and discusses the difference between the clustering algorithm and RPCA.